CHAPTER 3 Getting Statistical: A Short Review of Basic Statistics 45

Power, sample size, and effect size relationships

The α level of a statistical test is usually set to 0.05 unless there are special con-

siderations, which we describe in Chapter 5. After you specify the value of α, you

can display the relationship between α and the other three variables — power,

sample size, and effect size — in several ways. The next three graphs show these

relationships for the Student t test as an example, because graphs for other sta-

tistical tests are generally similar to these:»

» ^{Power versus sample size, for various effect sizes: For all statistical tests,}

power always increases as the sample size increases, if the other variables

including α level and effect size are held constant. This relationship is illus-

trated in Figure 3-2. “Eff” is the effect size — the between-group difference

divided by the within-group standard deviation.

Small samples will not be able to produce significant results unless the effect size

is very large. Conversely, statistical tests using extremely large samples including

many thousands of participants are almost always statistically significant unless

the effect size is near zero. In epidemiological studies, which often involve

hundreds of thousands of subjects, statistical tests tend to produce extremely

small (and therefore extremely significant) p values, even when the effect size is

so small that it’s of no practical importance (meaning it is clinically insignificant).»

» ^{Power versus effect size, for various sample sizes: For all statistical tests,}

power always increases as the effect size increases, if other variables including

the α level and sample size are held constant. This relationship is illustrated

in Figure 3-3. “N” is the number of participants in each group.

FIGURE 3-2:

The power of a

statistical test

increases as the

sample size and

the effect size

increase.

© John Wiley & Sons, Inc.